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ABSTRACT 

'This aanual briefly reviews the major guidelines icor 
a 'Title I evaluation plan using the ncra-referenced evaluation model. 
Model A*1.- Designed to help local school district personnel 
^responsible for Title* I evaluation activities^ th^ manual can be used 
as a quick check of present evaluation activities or' as a planning 
guide £cr future activities. The manual is divided into three 
sections- in the first. Title I Model A Testing Plan, a form is 
provided for recording Significant information aDout a testing plan* 
In-the second section. Title I Evaluation Guidelines, a series of' 
clipped pages with arrows at the edges 'tefer the reader zo certain 
sections of the- testing form* The pages^ provide information about the 
following guidelines: how using Mqdel^' measures the impact of Jfitle 
I programs, separation of pretest and selection, test forms and 
levels, testing at empirical norm* dates, and measuring sustained 
effects* Additional information is presented in the thira section. 
Tl^;is refers to ideas mentioned in the guidelines wnich provide more 
information on concepts 'involved in Title. I, evaluation* Inis section 
includes noraal curve equivalents,; why selection and pretest should' 
be separate, functional-level testing, vhy it's important to test at 
norm dates, and Title I evaluation technical standards^ 
(Author/GK) 
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MODEL A 
OWNER'S MANUAL 



Developed by; 

Randy Derpaline 
Don Rader 



' Distributed by: 

Title I Technical Assistance Centers 
Regions 8, 9 and 10 
Northwest Regional Educational Laboratory 
710 S.W. Second Avenue 
Portland. Oregon 97204 




Piirpose of Owner'I^Manual 



This manoal bricflv reviews the major guidelines for a Title I 
evaluation plan using the norm-referenced evaluation model, 
Model A-1 Specifics about Model A-2, using a m>n-normed 
test, are not covered in this manual. 

Designed to he||p l(Kal school district personnel responsible for 
Title I evaluation activities, the manual can bejused as a quick 
check of present ev aluation activities or as a planning guide for 
future activities 
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Description of Contents 

This manual is divided into three sections: 
Title I Model A Testing Plan 

A form is provided on page i\ for recording significant infor- 
mation about )our present or anticipated testing plan. If you 
want to use the form for planning the evaluation, you should 
duplicate it since a separate form is needed for each project and 
for different grades within each project. 

Title I Evaluation Guidelines 

A series of clipped pages foUow the Testing Plan form. Arrows 
at the edges of these pages refer you ^o certain sections.of your 
testing form. The pages provide information about the follow- 
ing guidelines: ^ ^ 

How Using Model A Measures the Impact of Your Title I 

Program T 

Separktion of Pretest and Selection . ..^ \ . . .2 

Testiporms and Levels . . *4 * ' 

Testing at Empirical Norm Dates .6 

Measuring Sustained Effects / : 8 ' 

Additional Information 

sThe rest of the manual contains sections providing additional 
information about some of the ideas mentioned in the guide- 
lines. They may provide a more in-depth rationale for the 
guidelines or more information on concepts involved in Title I 
evaluation. The) not meant to be technical explanations nor 
detailed steps in the implementation. If you need moce infor- 
mation, contact your Service Center identified on page 21. 

Normal Curve Equivalents ^10 

Why Should the Selection and Pretest Be Separate? 11 

Functional-Level Testing 14 

Why It's Important to Test at Norm Dates : . .16 

Title I Evaluation'Technical Standards 19 
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Title I Model A Testing Plan 



for. 



project r^me 



grade 



The folloAing information should be completed for each project and for each grade 
level within a project. 

Student Selection Information 

\ 

List any subtest names th^t be used in selecting students for Title I instruction 
(Selection of eligible students for Title I instruction may involve more than one test 
and other non-test data that have been identified as useful,) 



subtest name 



. from 



name of test 



-Date administered to students 



detest lnfoj[matiQn 



subtest name 



Subtest name from 



name of test battery 
-Date administered to students 



-Empirical norm date (see test battery manuati 
Level(s) of the test you are using 

1 Form of test you are using 



Posttest Information 

( 



subtest name 



-Subtest name f rorn _ 



name of test battery 
^Oate admmis^red to students 
Empirical norm date 



_Level(s) of the test you are using 



Sustained Effects Information 



-Form of test you are using 



subtest name 



-Subtest name from- 



name of test battery 
-Date {month and year) of administration 
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Title I Evaluation Guidelines 

HOW USING MODEL A MEASURES THE 
IMPACT OF YOUR TITLE I PROGRAM . 

To Whom Are Title I Students Compared? 

Your Title 1 students are compared to a na- 
tional sample of students who score at the 
same pretest percentile. For example, if your 
Title I class scores at the 25th percenfile at the 
pretest, its growth will be comparecf^o the stu- 
dents in the norm group who scored at the 
25th percentile. 

, • Estimating How the Students Would Have 
Don^ Without Title I Help 

ModeI\A's assumption is that a 'group of stu- 
dents w ho do not receiv e Title I instruction \v ill 
maintain a constant percentile throughout a 
' year. For example, a group of non^'iSc 1 stu- 
dents who scored at the^25th percentile at the 
beginning of the >ear is exj>ected to maintain 
the same relative standing (25th percentile) 
throughout the >ear. The achievement level of 
these students would incj^ease but so would the 
achievement levels of students above and be- 
low the 25th percentile). 

The Definition of^Title I Impact" 

Any change in the percentile rank gf aar 
of Title I students from the pretest to tflp 
V test is attributed to^le I instruction. Thus, it 
^ Title I students scored at the 25th jpercentile at 
the pretest and at the 30t^i per^ntile at the^^ 
posttest, the increase is considered to be the 
Title I impact. If the Title I students scorq^^t the 
same pretest and posttest percentile, the pro-^ 
gram would be considered to have hacf nom^ 
pact, students who receiv ed-4frrfy'*regular in- 
struction with no special help are expected to 
/ maintain the same percentile. , 
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SEPARATION OF PRETEST AND 
SELECTION 

Guideline 

The score used for selection should nothc used 
as the pretest score, > 

Purpose of the Guideline 
*> 

If the same test score is used for both selecting 
Title I students and as a pretest score, you will 
overestimate your project's impact. That is, 
your Title I program will appear to be more 
effective than it really was. 

Alternatives for Separating Pretest and 
Selection 

1 . Administer a separate pretest and selec- 
tion test^ 

2. Use last year's posttest scores as this 
yearns selection scores. 

3. Use'different subtests of the same test '^^ 
battery; one subtest for selection and the 
other for evaluation. Take care that the 
two1t€sts are related and both should be 
related to the objectives of the project. 

4. -Readminister the same test for the pretest 
•as was used for selection. 

i 

You should ev aluatc y our selection procedures 
at,th& end of each year to make any adjust- 
ments or changes. " 

* Exception to the Guideline 

The pretest score can be used for selection only if. 

1. Selection is based totally on the pretest score such 
that the students assigned to the Title /program are 
those who score lowest on the pretest, and 

2. the correction formula in the Ubcr'b (Si/idc is used to 
adjust (i.e. correct) the group's pretest mean. The 
adjusted pretest mean is then used to estimate the 
expected posttest percentile rank of the group. 



o 7 8 



ERIC 



NOTES 



For more inkrrmation 
on separation of 
pretest an^ selection 



/ 

ti^n to page 11 



TEST FORMS AND LEVELS 
Definitions 

Different LEVELS of a test contain items of 
differei^Tdifficfulty. ' / • % 

Different FORMS ol a tesUvillbe at the same 
level of difficulty, but v\ill contain different but 
coqiparable itefeis. , 

Levels Gurdeline; ' 

Administer test levels whose difficulty and' , 
coHtent match ^e performance levels of the . 
students (functionaJ-le;^el testing). 

Forms Guideline 

Use the same test formS for pretest and post- 
test that the publisher used vvith the norming 
group. ^ ^ 
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NOTES 
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For more information about 
functional-level testing 



turn to page 14 



^ TESTING AT 
EMPIRICAL NORM DATES 



Guideline , 

Testing should occur within two weeks before 
or after the publisher's empirical riorm dates. 



Empirical norm dat^s are the dates that the 
test publisher actually administered the test 
,to a national sample. This is as^opposed to 
projected norms, where the norms are merely 
estimates. 

Purpose of the Guideline 

The norm group is the comparison grbup for 
Model A. To make accurate comparisons, the 
Title I group should be test^ at the sahie time 
of the school year as the norm group. ) 

Deviations from the norm dates shoul(M)e in 
the same direction and magnitude for both 
pretest and posttest. For example, if pretest- 
ing occurred one week before .the norm date, 
then posttesting should also occur one week 
before the norm date. 

Exceptions to the Guideline 

It IS allowable to deviate up to six u eeks either side of the 
norm dates0you decide to do ^o, you u ill need to per- 
form extra chmputations to adjust the evaluation results 
to validly reflect your project's impact. ^ 



Pefinition 
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MEASURING SUSTAINED EFFECTS 

Measuring*for sustained effects is not an offitial^compo- 
nent of Model A, It is a significant and required part of 
^ur evaluation plan and so has been included in this 
manual, ^ 

Guideline 

A follow up mcasur^or sustained effects must 
be administered after the posttest and at least 
twelve months after the pretest. 

Purpose of Sustained Effects Information 

It is important that gains made by students in 
Title I programsXfrom pretest to posttest be 
sustaiTi^G4 t>ver lonWr periods of time. Thus, a 
fnllowup me^SureAaftet* the initial pretest- 
posttest cycle is needed. How soon after the 
posttest the information should be collected 
depend^ on what questions you want to ex- 
amine (see page 9). This sustained effects in- 
formation must be used in subsequent pro- 
gram planning. 

What to Use to Measure Sustained Effects ^ 

The instrument used to measure sustained ef- 
fects must be an objective measure of educa- 
tional acWev ement in the basic skills and in the 
same content area as the projectf. 
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DESIGNING YOUR SUSTAINED EFFECTS STUDY 

Mot everyone's sustained effects study will be the same. Each 
project may look at different students at different times to see 
wjiat happens to students after the initial pretest-posttest*re- 
sultS' reported in Model A. 



NCEs 



Do students 
continue to improve'^ 



^ Maintain the 
poSttest level''' 

Drop off? 
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Pretest Posttest Sustained 

Effects 

^ '* Measure 

— ► 

This time period must be at least 12 months 

Three possible questions you might ask are: 

1. Are achievement gains occurring during the school year 
maintained over the summer^monthsl 

2. Do the effects of Title I ir^struction continue after the stu- 
dents leave the programf^ 

How are students per^fji'rtfilTig w ho w«re in Title I last year 
* and continued in the Title I program? 

lotice that these three questions differ ^n: 

'horn to Measure — ^udents sfill in the Title I project or stu- 
dents who are no longer receiving services. 

* . ^ . 

When to Measure — Question 1 would imply a spripp posttest 

and a fallsb^tairiex^^fects measure whereas Questions 2 and 
3 imply the collection of data the following spring. 

Piggybacking Sustained Effects Measure With Model A 
Testing 

Collection of sustained effects information do^^s not zdways re- 
quire additional testing. For example, a pretest foMjne year 
could also serve as a measure of sustained effects for a previous 
year. , • \ 
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Additional Information 

NORMAL CURVE EQUIVALENTS (NCEs) 

Yuu might find it cas\ tu use percentile ranks to interpret the 
gains made by a group of Title I students. For example, >our 
Title I cla^s ma\ hav e progressed from the 1 5th percentile at the 
pretest to the 18th percentile at the posttest. 

However, there are problems with using percentile gains. They 
cannot be^sed for comparing jour Title I group's gain to other 
Title I groups. Thev also cannot be used for combining project 
gains across projects (as in a district report). Percentile gains 
have different meanings at different places^ on the percentile 
scale. In the figure below, you will see that a gain of 10 percen- 
tiles at the high or low ends^ the scale is much larger than a 10 
percentile gain in the middle. fSe<? the shaded bars.)' 

To avoid the problems with percentiles, NCEs were constructed 
to have equal units along a scale. Percentiles and NCEs match at 
the 1st, 50th, and 99th points on the scales. The two scales laid 
adjacent toj^ne another look like this: 

Normal Curve Equivalents 

t v !i H i!i!i- M t'ii|i|ii i l ml H '|i " 'i!;f'i'iii'!iii i iiiii> M it M !i.i>i' | ri ' 

1 10-20 30 40 50 60 70 80 9P 99 

m 

Percentiles 

1 10 30 50 70 90 99 

K 

' .V ■ . 

,You can see that a gain of 10 NCEs is the same an>where, alofig 

the *>cale. So by ^sing NCEs, )ou can more accurately ciimpare" 

the gains of >our Title I students to those in'other Title I^rojects. 

An Abbreviated Percentile to NCE Table 



%ile 


-trtE 


%ile 


NCE 


%fle 


NCE 


1 


1 0 


35 


4% 9 


70 ^ 


61 O""' 


5 


15 4 


40 


447 


75 


642 


10 


23 0 


45 


47 4 


80 


67,7^ 


15 


28 2 


50 


50.0 


85 


71.8 


20 


32 3 


55 


52 6 . 


90 


77 0 


25 


35 8. 


60 


55 3 


95 


84.6 ^ 


30 


39 0 


65 


58.1 


99 


950 ^ 



WHY SHOULD THE SELECTION ANI> PRETEST 

BE SEPARATE? ^ 

Weliave stated that the selection *and pretest should be^epa- 
rate. We also have said that failure to separate them will result 
in an o\erestimatio;i of project impact. The following explana- 
tion shows how this happens. 

Test scores rahk students on an achievement scale. 



ACHIEVEMENT SCALE 



Low 



a student's score 



High 



If you gave a test to all the students in a grade, you might expect 
to'flnd results similar to the figure below — most of the students 
would score around the middle with a few stude;its at the hijgh* 
and low ends of the scale. 




LOW 
. Some low 

^ achieving ^dents the middle range achieving students 

You may use a test like this to select students for Title I instruc- 
tion. - • ' . 




LOW , . HIGH 

Lowest score group Title I 




If you gave thfe same test to these Title I students the next day. 
would you expect the students to score the same? 

Probably not. Some students would score higher, some, lower. 
The resulting curve from the second testing would be more 
spread out than the curve from the first testing. 



, First Testing 
.Second Testing 



Low 



High 



What has happened to the average score of the group between 
the two testings? 



The average score on the 
Second Testing 




is HIGHER THAN 



the average score on the 
First Testing 




Average ' 



But this average is higher simply because of measurement 
error. ^ . 
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How does this artificial gain (tcchnicall) known as regression to^ 
the mean) affect Title I evaluation results? 

In Title I evaluation, a group*s achievement gain is found by 
comparing its pretest average with its posttest average. 

If you used the group*s selection average as its pretest average, 
you would be including this artificial gain in the estimate of the 
impact of your Title I project. 



Selection Test 



Pretest 




Posttest Average 



Selection 
Test Average 



when Selection Te^r 
used as pret^ 




The actual gain shown^ji;^>rU^^^ is the gain frofnthe pretest 
average to the posttest^v^e^^a^ * 

The apparent gain w*ould]3^c^}le gain firom the selection test 
average to the postttst kv^Hge. It is a combination of the actual 
gam plus the artificial gain. You can see that it is an^overestima- 
tion of the project's impact., , 
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FUNCTIONAL-LEVEt TESTING 

"what Is It? 

> 

Giving a student a test lev el that has been recommended bvthe 
.publisher fur the student's i?rade level is^alled in-le^el testing. 
Huvvevel*. the recommended level uf a test dues nut always cun- 
tain the must apprupriate cuntent ur difficult) fur students with 
very luw ur very high perfurmance levels. When, testing such 
students^yuu may want tu administer a test level uther than the 
specific level recummended by the test publisliet* fur typical 
students in that grade. Testing with an easier ur mure difftcult 
level than recummended by'the publisher is called ou{-of-level 
testing. Whenever you give a test level that you feel is most 
apprupriate fur a student's perfurmance level, it is cunsidered 
functional-level testing. 

Why Do It? / ^ 

Must students' functiunal levels are bes/ served when the test 
publisher's recummended level is admimstered. This will usu- 
ally result in a valid measure uf the achievement uf a gruup of 
students. Huvvever a test that is too difficult or too easy may 
provide very little information about students' actual achieve- 
ment. Students who are frustrated by a test that is too difficult 
ma^ive up early, or they may simply guess their way through 
the test. If a test'is too easy, students frequently lose interest. In 
either case, the test scores will not provide a valid or reliable 
assessment of achievement. 

* ^ 

Testing at the functional level is especially important in Title I 
evaluation because mostTitle I children areitHv^^a^evers, They 
might be tested more effectively with a lower level than recom- 
mended for most students in their grade. Obtaining v alid meas- 
ures of the group s ability both at the pretest and post^est will 
provideva more v alid measure of the Title I pri^ject's impact. 
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What You Need for Out-of-Level Testing 

If you are planning. to use out-of-le\el testing, your test must 
meet some necessary criteria. 

1 . The Test Must Have an Expanded Scale Score. To be able to 
compare students taking different test levels, test pub- 
lishers provide a \\ ay of placing all students on a common 
scale regardless of what level tbepvere administered. The 
score from this scale is geperalj^^own as an expanded 
scale scyre. The exact name nray differ from publisher to 
publi&Ker. ^ 

If youfare consideg<|i»g^t-of-le\el testing, l6ok in your test 
manual and find the name of the expanded scale score. 
Write it here: 



Expanded ScaJe Score Name 

2. You Must Be AblHv Follow the Subtest Used Across the 
Levels. ThG subtest Ascd for evaluation on the test level 
administered must also^ppear on the recommended test 
leveL For example, if the "Word Analysis" score is used on 
the out-of-level test, it must also appear on the in-level test. 
Check: . ^ 

What test level arc you going to administer? 

Whaj subtest score are you using from this test? 

What test level does the publisher recommend? 

Docs the subtest score occur on both the recommended and 
the administered levels? If not, this score canot be used for 
out-of-level testing. 

3. Norms Must Exist for the^Students' Grade LeveL To be .able 
to use the test in Title I Model A evaluation, norms must 
exist for the grade level for the^^tudents. That is, even 
though you ma> be giving 8th graders a level designed for / 

\ 6th graders, norms still ftiust be available for 8th graders. / 
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WHY ITS IMPORTANT TO TEST AT NORM DATES 



Testing at empirical norm dates is important if >ou expect to 
obtain usable data from \our Model A evalCiation. Let's con- 
sider the cases where the pretesting occurs at, before, or after 
the norm date. 

Case 1 : Pretesting at the Empiricat Norm Date . 

If you test at the norm date,\our Title I students will have 
received approximatelj the sany amount of instruction as the 
norm group did before being pretested. Your Title I group's 
performance on the pretest is compared to the norm group's 
performance and is reflectedHn theipercentile rank. Any per- 
centile increase at the posttest time \h the program's impact. 

Example: Suppose \ou oretes^OG^ Title I group of students 
at the empirical norm daic-5Kfa7ound that as a group they 
scored at the 30th percentile. If at the posttest the group 
scored at the 40th percentile, )ou would be correct in stat- 
ing that your Title I program increased the group's stand- 
ing from the 30th to the 40th percentile. 



Pretest 



Posttest 




Testing at Norm Date 



Posttest Average 



I 



Actual Impact 
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Case 2: Pretesting Before the Norm Date 

If you pretested your students too soon before the norm date, 
your students would have consid^erabl> less instruction than 
students in>he norm group had before they were, pretested. 
They would not bb expected to do as well as the norm group and 
they would score at a /oit'er percentile. 

. Example: Using the same group of students as in Case 1, if 
you pretested considerably before the norm date, they 
would score lower than the 30th percentile, say the 20th 
^per/rentile. If you compared that with the posttest average 
of the 40th percentile, you would be overestimating the 
impact of the Title I program.by saying it increased the 
group^s standing from the 20th to the 40th percentile. 



Pretest 



Testing Before Norm Date 



Posttest 




Posttest Average 



Overestimation of lm*pact 
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Case 3 : Pretesting After the Norm Date • f 

If you tested >our students too far after the norm date, your 
students would have had considerablv more instruction than 
students in tTie norm group had befi)rc the> were pretested. 
They w^)uld probablv score much higher than if tested at the 
norm date. 

Example: If you pretested the students in Case 1 consider- 
ably after^e norm date, they would score higher than the 
30th percentile, i>ay the 35th percentile. If you co'^mpared 
that with the posttest average of the 40th percentile, you 
would be a/zc/erestimating the impact of the Title I 
program, - f 



Pretest 



Testing After Norm Date 



Posttest 




Underest I matron of Impact 



The results would be even more complicated if the posttesting 
did not occur at the norm date. 
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TITLE I EVALUATION TECHNICAL STANDARDS 



For your Model A evaluation to run well you must follow certain 
maintenance requirements. In Title I evaluation, these require- 
mejits are known as the Technical Standards. 

1. Representativeness of evaluation findings. The evaluation 
results must be computed so that the conclusions apply to 
the persons or schools served by the Title I project. This 
may be accomplished by including in the evaluation either 
ail or a representative sample of the persons or schools 
served by the project. * - 

2. Relia'ditii\and;validity of evaluation instruments and 
procedures. The proposed evaluation instruments: 

" a. Must consistently and accurately^asure the obje^^ 
tives of the project; and. ' ' ^ 

b; Must be appropriate, considering factors such as age 
or background or the persons served by the project. 

3. Evaluation procedures'^ minimize error. Th^ proposed 
evaluation procemJreTnTimi^^ error by including: 

a. Proper administration of the Valuation instruments; , 

b. •Accurate scoring and transcription of data; and 

c. Use of analysis procedures whose assumptions are 
appropriate for the data. 

4. Valid assessment of achievement gains in heading, Ian- , 
guage arts and mathenjatics. In a&sessin&the effectiveness 

^_of regular school year Title I reading, language arts, and 
mathematics prQjects in grades 2 throu|;li 12. the pupoposed 
•evaluation procedures yield a valid measbt^o^ljZhe Title I 
children's performance after receiving Title I services com- 
pared to (2) an estimate of what t*ieir performance would 
have been in the absence of Title I services. 

If you need more information aboi/t these standards-, contact 
your Service Centers listed on page 21. /' \ ^^^^ 
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A WARNING ipROM THt MODELS ENGINEERS 

Goud drivers know the limitations of their automobiles asgood^ 
evaoiators know the limitations of tht^e\aluati«>n designs. The 
information gained from a Model A evaluation is limited in that 
it consists only of achievement data. Overinterpreting these 
results without considering other special programs or data such 
as attitudes, different achievement measures, and costs (dollars 
and time) vv/iuld be inappropriate. 

• ' . >) 
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SERVICE CENTERS 

For further assistance with your Model A evaluation, contact 
your authorized ser^^ice representatives : 





State tf^tacti^ame * 








^ 

Address 


m 




' • ^ . ' Phone Number 



m 

Northwest Regional Educational Laboratory 
Title I Evaluation Technical Assistance Centers 
710 S.W. Second Avenue 
Portland, Oregon 97204 
(503) 248-6971 

r 
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